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Reply to the Office Action of June 14, 2005 

Amendments to the Claims: 

This listing of claims will replace all prior versions, and listings, of claims in the application: 

Listing of Claims: 

Please amend claims 1, 12, and 18 as follows: 

1 . (Currently Amended) A method of recovering depth information for pixels of a base image 
representing a view of a scene, the method comprising the steps of: 

detecting a plurality of pixels in a base image that represents a first view of a scene^ 
wherein the first view is one of at least three views of a scene; 

determining 3-D depth of the plurality of pixels in the base image by simultaneously 
matching correspondence to a plurality of pixels in a plurality of imag e s three or more images , 
each image representing one of the at least three views of the scene, wherein each of the at least 
three views of the scene are situated in a non-linear arrangement and are further oriented in a 
plurality of non-parallel planes relative to each other; and 

tracing pixels in a virtual piecewise continuous depth surface by spatial propagation 
starting from the detected pixels in the base image by using the matching and corresponding 
plurality of pixels in the plurality of three or more images to create the virtual piecewise 
continuous depth surface viewed from the base image, each successfully traced pixel being 
associated with a depth in the scene viewed from the base image. 
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2. (Previously Presented) The method of Claim 1, wherein the step of determining the 3-D depth 
comprises the steps of: 

detecting a plurality of image pixels in a first image corresponding to a first view of a 

scene; 

detecting a plurality of image pixels in at least a second image corresponding to a 
respective at least a second view of the scene, wherein the at least a second image deviates from 
the first image as a result of camera relative motion; 

determining a first two-view correspondence between the plurality of detected image 
pixels in the first image and a plurality of detected image pixels in one of the at least a second 
image resulting in a potential two-view match set of candidate image pixels between the first 
image and the one of the at least a second image; 

concatenating a plurality of two-view match sets to form a first multiple-view potential 
match set; and 

determining a second multiple-view potential match set that is a refinement of the first 
multiple-view potential match set. 

3. (Previously Presented) The method of claim 2, wherein the second multiple-view potential 
match set is based at least in part on a least median of squares computation of the reprojection 
errors related to matched pixels in the first multiple-view potential match set. 

4. (Original) The method of Claim 1, wherein the tracing step comprises the step of: 

propagating a front of a virtual piece of a continuous depth surface to at least one 
neighboring pixel starting from the detected pixels in the base image. 

5. (Original) The method of Claim 4, wherein the tracing step comprises the step of: 

determining when a boundary is reached between two propagating fronts of virtual pieces 
of a continuous depth surface. 
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6. (Original) The method of Claim 4, wherein the tracing step comprises the steps of: 

comparing the matching costs of the two propagating fronts about the reached boundary; 

and 

stopping the propagation of the front with the higher compared matching cost. 

7. (Original) The method of Claim 1, wherein the tracing step comprises the step of: 

propagating a front of a virtual piece of a continuous depth surface to at least one 
neighboring pixel surrounded by a predefined size window in the continuous depth surface. 

8. (Original) The method of Claim 7, wherein the tracing step comprises the step of: 

determining when a boundary is reached between two propagating fronts of virtual pieces 
of a continuous depth surface. 

9. (Original) The method of Claim 7, wherein the tracing step comprises the steps of: 

comparing the matching costs of the two propagating fronts about the reached boundary; 

and 

stopping the propagation of the front with the higher compared matching cost. 

10. (Previously Presented) The method of Claim 9, wherein the matching cost is determined by 
computing the summation of all the normalized cross-correlations between a first image window 
of a pre-determined size in the base view, and a second image window of the same pre- 
determined size in one of at least one reference view. 

1 1 . (Previously Presented) The method of Claim 1 0, further comprising a step of: 

rectification of at least one pair of images corresponding to the base view of the scene 
and one of the at least one reference views. 
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12. (Currently Amended) An image processing system comprising: 

a memory; 

a controller/processor electrically coupled to the memory; 

an image matching module, electrically coupled to the controller/processor and to the 
memory, for providing a plurality of seed pixels that represent 3-D depth of the plurality of 
pixels in the base image view of a scene by simultaneously matching correspondence to a 
plurality of pixels in a plurality of three or more images representing a plurality of views of the 
scene, wherein the base image view is one of at least three views of the scene, each of the at least 
three views of the scene being situated in a non-linear arrangement and are further oriented in a 
plurality of non-parallel planes relative to each other; and 

a propagation module, electrically coupled to the controller/processor and to the memory, 
for tracing pixels in a virtual piecewise continuous depth surface by spatial propagation starting 
from the provided plurality of seed pixels in the base image by using the matching and 
corresponding plurality of pixels in the plurality of three or more images to create the virtual 
piecewise continuous depth surface viewed from the base image, each successfully traced pixel 
being associated with a depth in the scene viewed from the base image. 

13. (Original) The image processing system of claim 12, further comprising at least one camera 
interface, electrically coupled to the controller/processor, for sending image information from at 
least one camera to the controller/processor. 

14. (Original) The image processing system of claim 12, wherein the controller/processor, the 
memory, the image matching module, and the propagation module, are implemented in at least 
one of an integrated circuit, a circuit supporting substrate, and a scanner. 
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15. (Original) The image processing system of Claim 12, wherein the propagation module is 
further for determining when a boundary is reached between two propagating fronts of virtual 
pieces of a continuous depth surface. 

16. (Original) The image processing system of Claim 15, wherein the propagation module is 
further for comparing the matching costs of the two propagating fronts about the reached 
boundary; and 

stopping the propagation of the front with the higher compared matching cost. 

17. (Original) The image processing system of Claim 12, wherein the propagation module is 
further for propagating a front of a virtual piece of a continuous depth surface to at least one 
neighboring pixel surrounded by a predefined size window in the continuous depth surface. 

18. (Currently Amended) A computer readable medium including computer instructions for a 3- 
D image reconstruction computer system, the computer instructions comprising instructions for: 

detecting a plurality of pixels in a base image that represents a first view of a scene, 
wherein the first view is one of at least three views of a scene; 

determining 3-D depth of the plurality of pixels in the base image by simultaneously 
matching correspondence to a plurality of pixels in a plurality of three or more images, each 
image representing one of the at least three views of the scene, wherein each of the at least three 
views of the scene are situated in a non-linear arrangement and are further oriented in a plurality 
of non-parallel planes relative to each other; and 

tracing pixels in a virtual piecewise continuous depth surface by spatial propagation 
starting from the detected pixels in the base image by using the matching and corresponding 
plurality of pixels in the plurality of three or more images to create the virtual piecewise 
continuous depth surface viewed from the base image, each successfully traced pixel being 
associated with a depth in the scene viewed from the base image. 
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19. (Previously Presented) The computer readable medium of claim 18, wherein the step of 
determining the 3-D depth comprises the steps of: 

detecting a plurality of image pixels in a first image corresponding to a first view of a 

scene; 

detecting a plurality of image pixels in at least a second image corresponding to a 
respective at least a second view of the scene, wherein the at least a second image deviates from 
the first image as a result of camera relative motion; 

determining a first two-view correspondence between the plurality of detected image 
pixels in the first image and a plurality of detected image pixels in one of the at least a second 
image resulting in a potential two-view match set of candidate image pixels between the first 
image and the one of the at least a second image; 

concatenating a plurality of two-view match sets to form a first multiple-view potential 
match set; and 

determining a second multiple-view potential match set that is a refinement of the first 
multiple-view potential match set. 

20. (Previously Presented) The computer readable medium of claim 19, wherein the second 
multiple-view potential match set is based at least in part on a least median of squares 
computation of the reprojection errors related to matched pixels in the first multiple- view 
potential match set. 

21. (Original) The computer readable medium of claim 18, wherein the tracing step comprises 
the step of: 

propagating a front of a virtual piece of a continuous depth surface to at least one 
neighboring pixel starting from the detected pixels in the base image. 

22. (Original) The computer readable medium of Claim 21, wherein the tracing step comprises 
the step of: 

determining when a boundary is reached between two propagating fronts of virtual pieces 
of a continuous depth surface. 
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23. (Original) The computer readable medium of Claim 21, wherein the tracing step comprises 
the steps of: 

comparing the matching costs of the two propagating fronts about the reached boundary; 

and 

stopping the propagation of the front with the higher compared matching cost. 

24. (Original) The computer readable medium of Claim 18, wherein the tracing step comprises 
the step of: 

propagating a front of a virtual piece of a continuous depth surface to at least one 
neighboring pixel surrounded by a predefined size window in the continuous depth surface. 

25. (Original) The computer readable medium of Claim 24, wherein the tracing step comprises 
the step of: 

determining when a boundary is reached between two propagating fronts of virtual pieces 
of a continuous depth surface. 

26. (Original) The computer readable medium of Claim 24, wherein the tracing step comprises 
the steps of: 

comparing the matching costs of the two propagating fronts about the reached boundary; 

and 

stopping the propagation of the front with the higher compared matching cost. 

27. (Original) The computer readable medium of Claim 26, wherein the matching cost is 
determined by computing a normalized cross-correlation between the windows associated with 
the two propagating fronts about the reached boundary. 

28. (Original) The computer readable medium of Claim 27, further comprising a step of: 

rectification of at least one pair of images corresponding to at least one pair of views of 
the scene. 
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29. (Previously Presented) A method of recovering depth information for pixels of a base image 
representing a view of a scene, the method comprising the steps of: 

tracing at least one parameter surface, each of the at least one parameter surface traced 
starting from at least one predetermined seed pixel point; and 

calculating a derivative of function E(g) with respect to parameter g by using finite 
difference to minimize the following equation 



// is a reference image, 
Iq is a base image. 

(«, v) is a pixel point closest to Pj(Co+gL(uj, v,)); 

Co+gL(Uj, vj) is a 3-D point that projects to (u j9 vj) in the base view, and to Pi{Co+gL(u j9 vj)) in the 
/ th reference view; 

Co is the base image camera's center of projection; 

L(upVj) is the unit vector from C 0 to the point on the image plane that corresponds the pixel 
(wy,v y ); and 

g is a depth parameter. 




where NCC[/,(w ? v)/ 0 (w ,v )Jis a normalized cross-correlation between /,0,v) and I 0 (u J9 Vj); 

(O 

I i (u 9 v) is a first window of size coxco centered at pixel (w, v) in 4 
I 0 (u J9 Vj) is a second window of size coxco centered at pixel («/,v,) in Iq, 
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